The standard error of measurement is a more appropriate measure of quality for postgraduate medical assessments than is reliability: an analysis of MRCP(UK) examinations

Background: Cronbach's alpha is widely used as the preferred index of reliability for medical postgraduate examinations. A value of 0.8-0.9 is seen by providers and regulators alike as an adequate demonstration of acceptable reliability for any assessment. Of the other statistical parameters, Standard Error of Measurement (SEM) is mainly seen as useful only in determining the accuracy of a pass mark. However the alpha coefficient depends both on SEM and on the ability range (standard deviation, SD) of candidates taking an exam. This study investigated the extent to which the necessarily narrower ability range in candidates taking the second of the three part MRCP(UK) diploma examinations, biases assessment of reliability and SEM.Methods: a) The interrelationships of standard deviation (SD), SEM and reliability were investigated in a Monte Carlo simulation of 10,000 candidates taking a postgraduate examination. b) Reliability and SEM were studied in the MRCP(UK) Part 1 and Part 2 Written Examinations from 2002 to 2008. c) Reliability and SEM were studied in eight Specialty Certificate Examinations introduced in 2008-9.Results: The Monte Carlo simulation showed, as expected, that restricting the range of an assessment only to those who had already passed it, dramatically reduced the reliability but did not affect the SEM of a simulated assessment. The analysis of the MRCP(UK) Part 1 and Part 2 written examinations showed that the MRCP(UK) Part 2 written examination had a lower reliability than the Part 1 examination, but, despite that lower reliability, the Part 2 examination also had a smaller SEM (indicating a more accurate assessment). The Specialty Certificate Examinations had small Ns, and as a result, wide variability in their reliabilities, but SEMs were comparable with MRCP(UK) Part 2.Conclusions: An emphasis upon assessing the quality of assessments primarily in terms of reliability alone can produce a paradoxical and distorted picture, particularly in the situation where a narrower range of candidate ability is an inevitable consequence of being able to take a second part examination only after passing the first part examination. Reliability also shows problems when numbers of candidates in examinations are low and sampling error affects the range of candidate ability. SEM is not subject to such problems; it is therefore a better measure of the quality of an assessment and is recommended for routine use

The role of patient experience surveys in quality assurance and improvement: a focus group study in English general practice.

BACKGROUND: Despite widespread adoption of patient feedback surveys in international health-care systems, including the English NHS, evidence of a demonstrable impact of surveys on service improvement is sparse. OBJECTIVE: To explore the views of primary care practice staff regarding the utility of patient experience surveys. DESIGN: Qualitative focus groups. SETTING AND PARTICIPANTS: Staff from 14 English general practices. RESULTS: Whilst participants engaged with feedback from patient experience surveys, they routinely questioned its validity and reliability. Participants identified surveys as having a number of useful functions: for patients, as a potentially therapeutic way of getting their voice heard; for practice staff, as a way of identifying areas of improvement; and for GPs, as a source of evidence for professional development and appraisal. Areas of potential change stimulated by survey feedback included redesigning front-line services, managing patient expectations and managing the performance of GPs. Despite this, practice staff struggled to identify and action changes based on survey feedback alone. DISCUSSION: Whilst surveys may be used to endorse existing high-quality service delivery, their use in informing changes in service delivery is more challenging for practice staff. Drawing on the Utility Index framework, we identified concerns relating to reliability and validity, cost and feasibility acceptability and educational impact, which combine to limit the utility of patient survey feedback. CONCLUSIONS: Feedback from patient experience surveys has great potential. However, without a specific and renewed focus on how to translate feedback into action, this potential will remain incompletely realized.This work was funded by the National Institute for Health Research Programme Grants for Applied Research (NIHR PGfAR) Programme (RP-PG-0608-10050).This is the final version of the article. It was first available from Wiley via http://dx.doi.org/10.1111/hex.1229

Educational supervision and the impact of workplace-based assessments: a survey of psychiatry trainees and their supervisors

<p>Abstract</p> <p>Background</p> <p>Educational supervision (ES) is considered to be an essential component of basic specialist training in psychiatry in the UK. However, previous studies have indicated variation in its provision, and uncertainty about structure and content. Workplace-based assessments (WPBAs) were introduced in 2007 as part of major postgraduate medical training reform. Placing considerable time demands on trainees and supervisors alike, the extent to which WPBAs should utilise ES time has not been specified. As ES and WPBAs have discrete (although complementary) functions, there is the potential for this increased emphasis on assessment to displace other educational needs.</p> <p>Methods</p> <p>All junior doctors and their educational supervisors in one UK psychiatry training scheme were surveyed both before and after the introduction of WPBAs. Frequency and duration of ES were established, and structure, content and process were ascertained. Opinions on usefulness and responsibility were sought. The usage of ES for WPBAs was also assessed.</p> <p>Results</p> <p>The response rate of 70% showed general agreement between trainees and supervisors, but some significant discrepancies. Around 60% reported 1 hour of ES taking place weekly or 3 times per month. Most agreed that responsibility for ES should be shared equally between trainees and supervisors, and ES was largely seen as useful. Around 50% of trainees and supervisors used 25–50% of ES time for WPBAs, and this did not appear to affect the usefulness of ES or the range of issues covered.</p> <p>Conclusion</p> <p>ES continues to be an important component of psychiatric training. However, using ES for WPBAs introduces the potential for tension between trainees' education and their assessment by emphasising certain training issues at the expense of others. The impact of reduced training time, WPBAs and uncertainties over ES structure and content should be monitored to ensure that its benefits are maximised by remaining tailored to individual trainees' needs.</p